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A NON-PARAMETRIC  ALGORITHM  FOR  PATTERN  CLASSIFICATION  by  Witold  Malina 


The  classification  algorithm  described  can  be  used  in  the  case  of  /483 
incomplete  initial  information.  A preprocessing  layer,  a trans- 
formation layer  and  a decision  layer  are  distinguished  in  the 
classifier  from  the  functional  point  of  view.  During  study  a rule 
was  applied  according  to  which  individual  classifier  layers  were 
later  adjusted. 

The  essence  of  the  algorithm  concerns  the  division  of  pattern  space 
into  cells.  Then  a histogram  is  devised  for  each  class,  an  estimate 
of  the  unknown  distribution  of  the  probability  of  a pattern  arising 
from  particular  cells.  The  construction  of  cells  and  histograms  takes 
the  form  of  initial  information  into  consideration.  The  details  of 
the  algorithm  are  explained  by  example. 

Part  Two  gives  an  idea  of  the  practical  implementation  of  the  basic 
unit  of  the  classifier  in  the  form  of  two  layers  of  adaptive  logic 
networks.  Each  layer  of  the  logical  network  can  be  composed  of 
ASL-3  modules.  The  ASL-3  module  (Figure  9)  is  the  basic  structural 
unit  of  the  classifier.  It  is  an  adaptive  logic  network  with  three 
inputs  and  one  output.  When  a classifier  is  built,  there  is  a possi- 
bility of  connecting  it  in  a simple  way  for  work  with  a computer. 

I.  Description  of  the  Algorithm 
1.  General  Considerations  and  Assumptions 

The  basic  part  of  the  classifier  consists  of  two  adaptive  logic 
network  layers.  In  connection  with  this,  all  input,  interlayer  and  output 
signals  can  have  the  form  of  binary  pulses,  in  order  to  enable  classification 
of  patterns  even  when  input  signals  y^,  y2 , ...  yN  are  not  binary  or  when  their 
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Figure  1.  Structure  of  logic  classifier.  Key:  1,  basic  part  (logic), 

2-pattern,  3-initial  processing  layer,  4-first  logic  layer,  5-second  logic 
layer.  (^~  Aeci^io* 

number  exceeds  the  input  number  of  the  basic  part  of  the  classifier,  a layer 
for  preprocessing,  composed  of  linear  elements  EL  and  analog- 

digital  A/C  [AD]  converters,  was  introduced.  On  the  structural  side  the 
number  of  dimensions  of  the  original  pattern  space  is  limited  by  the  number  N 
of  the  coeffic  ents  of  linear  elements.  From  the  functional  point  of 

view  we  can  distinguish  the  following  elements  in  the  structure  of  the  logic 
part  of  the  classifier  (network) : 

a)  Vertically,  two  layers  which  fulfill  the  functions  of  transfor- 
mation and  decision  register  respectively,  and 

b)  Horizontally,  L,  identical  layers  corresponding  to  discriminators 
in  other  classification  system'5.  A general  block  diagram  of  the  logic  layer 

classifier  for  L class  is  shown  in  figure  1.  / 484 
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Figure  2.  Examples  of  the  route  of  auxiliary  functions  for: 

a)  < . -*  1 . b)  f.  < 1 - c)  i>  < 

The  assumptions , on  which  the  working  algorithm  of  the  system  discussed 
is  based,  are  divided  into  two  groups. 

A.  Assumptions  About  Patterns 

1)  There  exists  an  absolute  N-dimensional  space  of  features  ... 
describing  the  properties  of  classified  patterns.  The  numerical  values  of  their 
parameters  can  be  discrete  (binary  in  particulat)  or  variable  in  a steadv  manner 
in  determined  sections. 

2)  As  a result  of  pattern  description  by  the  absolute  number  of  parameters 
and  of  their  quantification,  the  correspondence  between  pattern  and  numerical 
values  describing  parameters  is  not  ambiguous. 

3)  The  statistical  properties  of  patterns  and  of  parameters  describing 
them  are  unknown;  observations  may  have  a multimodal  distribution. 

4)  An  acquaintance  with  class  xj  is  based  on  given  sets  of  observation 
Y(x  ),  l = 1,  2,  ...  L. 

B.  Assumptions  Referring  to  the  Algorithm 

1)  Classifier  decisions  must  have  the  form  of  a binarv  route. 
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Figure  3.  Simplified  block  diagram  of  full  study t classification  algorithm# 
key:  1-data  set  studying  Y(x,),  l*  1,  2,  ....  L,  2-reduction  of  dimensions 


of  original  pattern  space,  3-construction  of  auxiliary  functions.  Breakdown 


of  pattern  space  into  cells  K , 4-creation  of  histogram 


evaluation  of  the  distribution  of  probability  P(y#K  ,x?)  for  a determined 


division  of  space  into  cells,  5-reading  of  pattern  y,  6-transformation  of 


pattern  to  reduced  space,  7-definition  of  cell  index  Kz  to  which  pattern  v 


investigated  belongs,  8-known  decision,  ^“desision  printout 
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Figure  4.  Structure  of  bidimensional  pattern  classifier.  Key:  1-pattem, 
2-first  logic  layer,  3-second  logic  layer,  4-decision. 

2)  It  is  assumed  that  the  decisive  rule  has  the  form 

x*(y)  = *i,  if  P(y  e Kx,  X|)  > P(y  e Kz,xj),j  = I,  2 ...  L.j  ± /.  (l) 

where  K is  the  z cell  (section)  occurring  as  a result  of  the  division 
z 

of  pattern  space  by  hyperplanes. 

3)  The  main  probability  of  classification  error 


L 

P = V P(z)[l  -maxP^lz)]  = V [V  P(:,  .x,)-ma  xP(:,  x,)j , 

1 i=i  1 


(?) 


is  adopted  as  a criterion  for  the  quality  of  the  decisive  rule, 

Where:  P(z)  is  the  probability  of  an  observation  of  y occurring  in 

the  z cell: 

P(x^/z)  is  the  apostenon  probability  that  <tn  observation  belonging 

to  the  cell  K is  an  element  of  class  x,; 

z 4 

p(z,x^)  is  the  total  probability  that  y belongs  to  a cell  of  index 
z and  is  an  element  of  class  x j. 

In  general  the  probabilities  occurring  above  are  not  known.  We  shall 

estimate  them  in  a study  cycle  on  the  basis  of  a study  set. 
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Figure  5.  Illustration  of  method  of  dividing  pattern  space  into  elementary 
cells  with  the  aid  of  auxiliary  functions. 

In  addition  the  algorithm  should  consider  aspects  of  structural 
realization  with  attention  to  the  possibilities  of: 

a)  construction  an  independent  system, 

b)  cooperation  with  any  electronic  computer. 

2.  Study  and  Classification  Algorithm 

The  classifier  study  cycle  is  based  on  the  given  sets  Y(x^),  1 =1;  . 
2, ...,L,  and  is  composed  of  two  stages: 

-Adjustment  of  the  preprocessing  layer, 

-choice  of  the  structure  of  two  logic  network  layers. 

In  the  study  cycle  we  made  use  of  a concept  of  successive  adaptation 
of  individual  classifier  layers.  A detailed  description  of  the  adaptation 

of  each  layer  will  be  given  below. 

2, ),  ./4^*praf?’o>7  M ftrecevzi’ry  iffer 

A need  for  the  existence  of  a preprocessing  layer  occurs  when 

the  number  of  vector  components  of  the  pattern  exceeds  the  input  number 
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Table  1.  Comparison  of  z routes  and  elementary  cells  corresponding  to  them. 


* 

* 

/ 

0 

0000 

1.  5,  9,  37,41,45,  73,  77,  81 

1 

0001 

6,  42,  78 

2 

0010 

10,  14,  18,44 

3 

0011 

15 

4 

0100 

2,  4,  8,  38,  40,  74,  76 

5 

0101 

3,  7,  39,  43,  75,  79 

6 

0110 

11,  13 

7 

0111 

12,16 

8 

1000 

28,  32,  36,  50,  54,  64,  68,  72 

9 

1001 

33,  51,69 

10 

1010 

19,  23,  27,  52,  55,  59,  63 

11 

1011 

24,60 

12 

1100 

29,  31,  35,  47,  49,  53.  65,  67,  71 

13 

1101 

22,  30,  34,  48,  66,  70 

14 

1110 

20,  26,  56,  58,  62 

15 

till 

21,25,57,  61 

I 


Key:  1-elementary  ctflls  eirWerin*^  composition  Kz. 


of  the  logic  part  of  the  classifier  or  when  the  signals  describing 
the  pattern  are  not  binary.  In  other  cases  we  can  omit  the  initial  layer, 
and  introduce  the  pattern  signals  directly  into  the  logic  part  of  the 
classifier. 

That  space  obtained  as  a result  of  the  transformation  made  by  the  initial  layer 
we  shall  call  reduced  space.  The  coordinates  of  this  space  are  identified 
successively  on  the  basis  of  research  on  the  mutual  overlap  of  study 
sets  Y(x^),  Jel,  2 ...  L. 

The  overlapping  of  the  sets  i6  described  by  the  matrix  [47,  or 
which  the  elements  /l  . are  a numerical  estimate  of  the  magnitude  of 

J 1 a 

covering  of  sets  Y(xi)  and  Y(x.)  . A decision  in  regard  to^successive, 

J 

additional  k coordinate  of  reduced  space  is  made  on  the  basis  of  examination 
of  the  breakdown  of  the  elements  of  the  sets  studied  along  the  coordinates 


mentioned  earlier. 
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Figure  6.  riguira  6 . Histograms  of  frequency  of  observations  occurring  from 
classes  x^  and  X2»  Parameter  £ = 0.7. 

As  a result  of  this  examination  the  elements  A.  jj,  for  which  the  relation^ 


Aij  ^ ^o>  i ^ j. 


(3) 


A 

where  Zj^is  the  given  threshold  value,  is  fulfilled,  are  replaced ^zeroes 


in  the  matrix  The  algorithm  completes  the  search  for  new  directions, 

if  all  the  elements  of  the  matrix  EdJ  were  made  equal  to  zero(then  k=k^; 
kQ  < I)  or  k=I. 

Finally  the  transformation  made  by  the  initial  value  of  N-dimensional 
space  in  the  reduced  kQ-dimensional  space  can  be  written  in  a matrix  form 


' yi 

an  •••  Gin 

>i ' 

= 

. y> o. 

«*„i  •••  ak0N. 

ys. 

(M 


where:  akj  is  an  element  of  the  matrix  transformation  sought,  while  (a^, 

a^p,  . . .,akjj)  are  a set  of  coefficients  of  the  linear  k element; 

y'k  is  the  k component  of  the  vector  of  pattern  y after  transformation; 


k=l,2, . . .,k^,kQ  I. 
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Figure  7-  Diagram  of  the  frequency  of  error  as  a function  of  parameter  £ . 
Key:  1-frequency  of  error. 

On  the  basis  of  the  above  equations  we  can  state  that  the 

processing  layer  plays  a role  in  the  system  adjusting  the  pattern  space 

to  the  number  given  ahead  of  time  for  the  I inputs  of  the  basic  unit  of 

the  classifier.  The  proper  choice  and  orientation  of  the  new  system 

of  coordinates  has  essential  influence  on  both  classification  mistakes 

and  on  the  degree  of  the  complexity  of  the  classifier.  However,  premises 

do  not  always  exist  which  permit  optimal  choice  of  the  number  of  linear 

v 

elements  into  their  parameters. 

2.2  Study  of  Logic  Layers 

Is  ^ by 

The  logic  (basic)  unit  of  the  classifies  form  two  layers  of  an 
adaptive  logic  network.  The  first  layer  of  the  logic  unit  transforms 
k^-dimensional  space  into  binary  space (the  k^L-dimension  as  high  as  possible) 
formed  by  the  apex  of  the  Superblotk  unit.  The  second  layer,  called  the 
decision  layer,  prearranges  a decision  in  conformity  with  the  rule  adopted 
for  each  apex. 

In  describing  the  transformation  mechanism  performed  by  the  first 
logic  layer,  it  is  convenient  to  make  use  of  the  set  of  full  auxilary 
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Figure  8.  Diagram  of  cooperation  in  the  cycle  studying  the  classifier  with  a 
computer.  Key:  1-pattern,  2-reduction  in  dimensions,  3-decision,  4-for  each  z 
e = 1,  5-computer. 

functions  D.  (y'£).  These  functions  are  constructed  in  consideration 
of  the  set  studied  on  the  basis  of  the  bnincli  functions  defined  below 


gkJ(y',e)  =G  W'-e} 


L 


where:  y£~ is  the  j observation  from  the  set  studied, 

u^  is  the  unit  vector  along  the  k coordinate  of  reduced  space, 

£ is  the  extrapolation  p^rame ter t £ (0, 1/ . 

G(t  )=€.,  where  t 0, 

|o,  where  t <.  0. 
k=l, 2, . . . , . 

With  the  aid  of  function  (5)  the  process  of  <i*tra  po\c*imy  the  property 
of  the  set  studied  is  fulfilled.  It  is  based  on  prearrangement  of  each 
observation  of  the  environment,  the  magnitude  of  which  depends  on  the 

perameter  £ . This  approach  is  a result  of  the  assumption  that 

[Continued  on  page  12].  3.0 
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Figure  9*  ASL-3  Module:  a)  Logic  diagrams  (b^_2,  bn-l’ 

input  signals,  s-module  selection  signal,  LP-memory 
elements);  b)  guide  matrix,  (d^-block  selection  signal  of 
class  1,  e-type  of  extrapolation,  Adr-signal  from  address 
network,  r^r^signals  implementing  extrapolation  process; 

c)  diagrams  illustrating  dependence  of  extrapolation  on 
parameter  e(Adr  signal  switching  on  LP4  derives  from 
observations  in  study  cycle). 

Key:  (1)  universal  logic  network;  (2)  deletion;  (3)  network 
addressed,  (4)  condition  LP^;  (5)  matrix  M. 


Table  2. 

Simplified 

table 

of  truth  of 

matrix 

M. 

( n ) Wejscia 

(2)  WyjScia 

e 

di 

Adr 

I r‘ 

LP 

n 

0 

0,1 

0,1 

0,1,2 

0 

0 

1 

1 

1 

0 

1 

0 

1 

0 

1 

1 0 

0 

0 

1 

1 

0 

0 

o 

0 

1 

0 

0 

0 

0 

0 

1 

1 

0 

1 

1 

-> 

0 

1 

0 

0 

0 

2 

1 

0 

1 

1 

0 

2 

0 

0 

1 

0 

0 

y 

1 

1 

0 

1 

2 

3 

0 

1 

0 

0 

0 

3 

1 

0 

2 

1 

1 

3 

0 

0 

0 

0 

0 

3 

1 

0 

: i 

1 

0 

Key:  1)  input;  2)  output. 


if  obserbation  Y[j]  belongs  to  Y(x,),  it  is  very  probable  that  an 
observation  which  occurs  in  the  working  cycle  in  the  environment 
y[j]  will  also  belong  to  this  same  class. 

We  obtain  the  auxiliary  function  DJk  as  the  logical  sum  of 
the  proper  extrapolation  functions 


Dlt(y',  e ) = sgnj^1  d,gkj(y',  e)]  ( 6 ) 

1 *(*) 


where  Y{x)  = Y(x,)  v Y(x2)  v ...  Y(x£," 
j where  y[j]  e Y(xt), 

‘ ~ ]o  in  the  opposite  case, 
/ = 1,2...  L. 


Functions  (6)  can  be  considered  as  a certain  form  of  the  binary 
description  of  the  proper  of  class  x.  along  the  k coordinate. 


Figure  10.  Logic  network  composed  of  ASL^3  modules, 
Key*  (1)  Matrix  S > (2)  pattern;  (3)  decision. 


An  important  moment  in  the  adjustment  of  the  first  logic  layer 
is  the  proper  choice  of  values  of  parameter  e.  The  value  of  this 
parameter  depends  on  the  properties  of  the  set  studied. 


Figure  2 shows  examples  of  the  roor •"&  of  auxiliary  functions 
constructed  on  the  basis  of  two  observations  for  different  values  of 
parameter  £ . 

In  particular  for£=l,  it  can  be  seen  (fig.  2a)  that  the  auxiliary 

function  assume4  a value  equal  to  unity  only  at  definite  points  corresponding 

Z 


to  the  values  of  the  k feature  of  observation^from  the  study  cycle.  For  t < 
the  auxiliary  function  assumes  a value  equal  to  unity  not  only  at 
the  definite  point,  but  also  in  a definite  environment  of  these  points 
(fig.  2b).  With  further  decrease  in  the  values  of  perameter  G , the 

environments  around  the  observation  will  increase  and  a situation  may 

6 -fO 

occur  as  seen  in  figure  2c.  If  f,  this  magnitude  of  environemnt  leads 

A w 

to  infinity. 

Since  the  auxiliary  functions  can  assume  values  Q or  1,  in  regards 


to  it  the  transformation  layer  of  the  logic  classifier  (fig.  l)  transforms 
the  k-diraensional  vector  y into  kQL-dimensionai^  binary  vector  z.  The 
dependence  between  these  vectors  can  be  written  in  the  form 


* = T(y), 


where  the  transformation 


(7) 


T(y')  = {!),,(/,  e),  ...  Dlk(y\  e),  ...  D«0(.v',  e)} 


(8) 


depends  on  £ , while  JSsk(y’  ) means  the  k auxiliary  function  of  the  1-class. 

It  is  convienent  to  explain  the  activity  of  the  transformation  layer 
by  calling  on  a geometric  interpretation.  For  this  purpose  let  us  assume 
that  hyperplanes  perpendicular  to  the  proper  axes  of  the  set  of  coordinates 
are  drawn  through  the  places  of  brwnelies  of  all  auxiliary  functions.  Then 


ll+ 


the  pattern  space  will  be  divided  into  certain  sections,  elementary  cells. 

These  cells  have  the  property  that  all  vectors  y,  which  belong  to  a selected 
elementary  cell,  will  be  converted  into  the  same  vector  z (apices  of 
koL-dimensional  superblocks).  In  general  the  number  of  elementary  cells  is 
greater  than  the  number  of  sectors  z.  In  this  case  several  elementary  cells  /4f?P 
can  be  associated  with  the  same  apex. 

Let  us  designate  by  Kz  a cell  composed  of  elementary  cells  to  which  the 
same  superblock  apex  number  corresponds.  Then  by  the  index  of  cell  Kz  we  can 
understand  the  apex  number  written  in  the  form  of  a binary  number  formed  from 
components  of  vector  z.  The  value  of  index  z in  the  tenth  system  will  be 
designated  by  z' . It  can  be  computed  from  the  formula 

L *0 

z'  = y y,A»o\<o2,'-i,‘o+*-1.  (q) 

J=1 *=1 


The  study  algorithm  of  the  second  logic  layer  makes  use  of  the  knowledge 
of  the  route  of  implementation  of  a variable  random  Z,  together  with  data 
about  the  pertinence  of  observations  to  corresponding  classes.  The  histogram 

A 

P(z,  x^),  l - 1,  2,  ...  L,  being  an  approximation  of  the  distribution  probabi- 
lity  P(y£  Kz,  x^)  is  delimited  on  the  basis  of  such  routes.  The  task  which 

occupies  us  here  is  based  on  empirical  evaluation  (for  all  z and  classes  x ) of 

l 


the  following  probabilities 


J p(y,  x,)Ay  = P(yeK.,  x,)  = P(z,  .v,),  ( 10) 


where  P(y,  x^)  is  the  function  of  probability  density  of  the  pair  (Y,  X)  , 
Y is  a variable  random  constant, 

X is  a variable  random  discrete  number. 
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We  may  accept  as  an  estimate  p(z,xj ),  the  frequencies  computed  on  the  basis 
of  sets  studied  according  to  the  equation 


PCx.)  = 


m, 

L 

T.  rn  , 


J=  i 


P(yeK,lx,)  = 


mL[K'} 

m, 


/*  "1 

where:  m / K V is  the  number  of  elements  of  set  Y(x7)  contained  cell  K , 

Z ' i'  A z 

ml  is  the  total^ elements  in  Y(x^), 

Z-=l>  2 y . . . , L. 

The  estimate  of  the  distribution  p(z,x^), 1=1,2, .. .,L,  represents  a 
histognm  which  can  be  taken  as  a discriminatory  function  of  the  classifier, 
that  is  (p  ^(z )=P(z,x^ ) . With  the  aid  of  this  histogram  it  is  possible  to 
define  the  logic  structure  of  the  decision  layer  and  through  it  to  implement  a decision 
rule 


**00  = *1.  if  fi(z)  > TjO).  J = 1.2,  L,j  =£  1, 

(ID 

where  the  vector  z is  joined  to  the  vector  y by  the  transformation 
described  earlier. 

Rule  ( 11 ) is  a rule  approximating  the  rule  of  greatest  probability, 

and  its  quality  is  characterized  by  means  of  frequency  of  error,  which  is 

an  estimate  of  mean  probality  of  error  (2). 

A 

The  activity  of  the  decision  layer  can  also  be  described  by  means 


h 


of  auxiliary  functions  (6).  For  this  purpose  the  variable  y should  be 
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replaced  by  z and  it  should  be  assumed  that fc  =1.  Treating  the  variable 

/ u vli 

vector  z as  a number,  we  bring^concepts  to  unidimensional  space.  In  this  1 

case  we  designate  the  auxiliary  functions  for  the  decision  layer  by  D^(z,l), 

1=1,2, In  conformity  with  rule  (ll),  the  set  of  signals  represented 
by  the  values  of  the  auxiliary  function";  at  the  output  of  the  decision 
layer  of  the  classifier  (fig.  l)  has  the  following  form 

D^(z,l)=l,  if  z corresponds  to  an  observation  of  the  l class, 

{ 

0 in  the  opposite  case, 

Z=l,2, ...,L.  (12) 

Figure  3 presents  a general  diagram  of  the  study  algorithm  along  with  the 
classification  algorithm. 

An  important  step  in  finishing  the  study  cycle  is  an  investigation 
of  the  quality  of  classification  in  two  cases: 

a)  for  observations  from  the  study  cycle, 'tnl 

b)  for  observations  from  the  set  doing  the  examining  (known,  4neji 
which  did  not  appear  in  the  study  cycle). 

If,  after  the  study  has  been  made,  the  classifier  gives  a bi-passing 
decision  (neutral),  then  it  is  most  probable  that  the  cells  into  which  the 
new  observation "falls"  is  empty.  The  following  may  be  causes  of  this: 

-the  study  set  does  not  fulfill  the  conditions  of  good  representation, 

of  o-irdtatt  ctr  £. 

-the  choice, -i  is  unsuitable  (for  extrapolation  of  the  properties  of  the 
set  studied). 

We  can  improve  the  quality  of  classification  if  we  repeat  the  study 
for  a new  £ or  continue  it  for  observations  mentioned  in  point  b).  The 

r 

process  of  continuing  study  occur*«|  after  the  proper  conclusion  of  the 

study  cycle  and  is  based  on  correction^the  frequency  histogram'  (without 

ure 

changing  f ) . In  accord  with  the  new  histograms  changes  in  the  structure 

1 7 


n 


of  the  decision  layer  arc  made . In  the  case  of  a change  of  perameter  <*"• 
the  study  embraces  two  final  layers  of  classifier. 

3.  Example  of  Multimodal  Classification 


The  algorithm  described  will  be  illustrated  by  the  example  of  a 
bidimensional  pattern  classifier.  The  study  sets  for  the  current  example 
were  chosen  in  such  a way  that  no  need  to  use  the  preprocessing  layer  develop* 

In  connection  with  this  we  can  omit  this  layer  from  our  consideration  and  adopt  the 
following  assumptions: 

l)  the  classifier  is  composed  of  two  layers. 

?)  L=2,  that  is,  two  classes  of  patterns  exist,  x^  and  x^. 

3)  the  patterns  subject  to  classification  are  described  by  a set  of 

values  of  two  features  y^,y0,  which  are  realizations  of  a variable  random 

number  within  an  unknown  probability  distribution. 

U-)  the  observations  of  each  of  the  classes  are  extended  in  such 

a way  that  they  can  be  divided  into  two  different  sub  sets  of  set>,  Y(x  ) 

v 1 

and  y(x2).  Sets  Y(x^)  and  Y(x^)  have  fifty  elements  each  and  are  both 
included  in  figure  5-  The  symbol,,  • " symbolizes  observations  f'rooiclass  x^, 
while  the  sum  symbol  ,,  ^observation?  froivi  class  x^. 

5)  the  decision  rule  has  the  form 


x*(y) 


jf,  for P{z,  x,)  > P(:,  x2), 
x2for P(z,x,)  < P(z,x2), 
Ofor P(z,  .v,)  = P{z,  v2). 


03) 


(13) 

6)  the  criterion  for  the  quality  of  the  decision  rule  is  the  mean 
frequency  of  error. 
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We  begin  a study  of  the  logic  layers  of  the  classifier,  the  structure 


of  which  is  shown  in  fig.  by  delimiting  the  set  of  auxiliary  function^  f or 
each  class.  These  functions,  the  values  of  which  are  arguments  for  the 
decision  layer,  will  be  delimited  according  to  equation  (6). 


Figure  5 gives  an  example  of  the  division  of  the  pattern  space  into 

elementary  cells  on  the  basis  of  auxiliary  functions.  The  diagrams  of  the 

3 F =. 

appropriate  auxiliary  functions  are  set  for  the  parameter  *0.7  and  are 

plowed  in  the  figure  of  both  coordinates  y ,y  . 

1 ^ e 

The  binary  values,  which  the  auxiliary  functions  assume,  are  arranged 
by  the  individual  elementary  cells,  in  the  form  of  ftfute.  z.  In  the 
example  under  consideration  the  following  rule  was  adopted  for  delimiting 
the  elements  of  route- z^z^z^z^,  where:  z^=D^^ (y,  r ), z^*D^^(y,  i ),z^= 

D^Cy,  z^*D  (y,  £ ).  Elementary  cells  with  identical  routes  z form 
the  cell  Kz . An  orderly  comparison  of  all  the  elementary  cells  from  fig. 5 
(the  numbering  adopted  in  the  figure  is  completely  arbitrary)  is  given  in  table 
1. 

The  study  cycle  routes  • of  the  classifier  ends  with  the  construction 

of  a histogram  P(z,x  ) 1=1,2,  defined  by  the  cell  K on  the  basis  of  the  study 

1 J z 

set.  Fig.  6 shows  the  histograms  formed  for  tnis  example.  On  the  basis 
of  the  histogram? it  is  possible  to  select  a decision  layer  structure  so 

c; 

that  the  output  signal  of  the  classifier  ha^  the  form: 


Dy(z,  1)  = 1,  D2(z,  1)  = 0 forP(z,.r,)  > P(z,x2). 

D,(z,  1)  = 0,  D2(z,  1)  = 1 for  P(z,x,)  < P(z,x2), 

D{(z,  1)  = 0,  D2(z,  1)  = 0 for  F(z,  x,)  = P{z,x2), 

• 
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Where  D^(z,l)Z=l,2  designate* the  auxiliary  functions  describing  the  decision 
layer. 

The  diagram  of  frequency  of  decision  error  p(£  ),  shown  in  figure 
7 leads  to  some  interesting  conclusions.  It  is  obvious  from  this  diagram 
that  the  frequency  of  error  diminishes  when  £ tends  toward  1.  This  is  caused  /¥?•< 

by  the  fact  that^asif  increases^  the  auxiliary  functions  of  the  transformation 

^CQTr’b-j'lCe  * 

layer  D (y,  £ ),l, k=l,2  assume  a "(fOnftr’  form.  The  dimensions  of  the  cells 
tk 

diminish  in  connection  with  this.  Consequently  the  possibility  of  occurence 
of  cells  containing  observations  of  both  classes  at  the  same  time  diminishes. 

= f 

A point  corresponds  to  the  border^/-  4k  of  each  cell  which  contains  at  most 

JsZh-c- 

one  observation.  This  example  is  useful  for  determini»g  situations  when 
pattern  space  element^  form  a final  set  of  points.  For  statistical  situations! /W-i 
when  observations?  can  form  an  infinite  set  of  point,  it  is  necessary  to  extropolate 
the  properties  of  the  classes  on  the  basis  of  study  sets.  For  this  purpose 
it  is  necessary  to  find  an  adequate C < 1. 

It  is  worthwhile  attending  to  the  fact  that  as  it  becomes  more 
distant  from  the  points  corresponding  to  the  hypothetical  modes  of  the 
curve  of  conditional  probability  density,  the  approximation  of  the  decision 
ranges  to  the  cells  Kz  is  worse.  At  the  same  time  this  does  not  have  a 
great  deal  of  effect  of  increasing  the  value  of  mean  probability  of  error, 

because  the  probability  of  drawing  the  signal  y ,y  from  these  areas  and 

1 2 
a 

qualifying  it  otherwise  than  would  result  from  the  decision  rule  adopted 
is  relatively  slight. 

The  algorithm  described  has  been  used  to  recognize  numbers  written 
by  hand.  Runs  of  binary  number* have  been  arranged  for  the  recognized 
number  in  such  a way  that  they  correspond  to  signals  from  photq  elements 
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of  the  receptor  field  of.  dimensions.  5x J elements.  Experi  ,®ental  research 
with^Algorithm  has  been  carried  out  on  the  ODRA  120^  computer.  Some  results 
of  this  research  are  discussed  in  A, 5-7. 
k.  Final  Conclusions 

Some  remarks  must  be  made  after  this  general  presentation  of  the 
algorithm  and  of  tests  using  it  for  classification  problems. 

One  fault  in  the  algorithm  lies  in  the  transformation  of  vector  y 

. I 

into  binary  vector  z.  The  faUtitWQl  number^  components  of  vector  z is  the 
product  of  tne  number  of  dimensions  N of  pattern  space  and  the  number  of 
classes  L.  In  this  connection  the  dimensions  of  vector  z will  be  large- 
for  jur^e.  N and  L.  Among  other  things  this  can  cause  prolongation  of  study 
time,  time  for  reaching  a decision,  and  difficulties  in  implementing  the 
algorithm.  In  regard  to  the  above,  ^reduction  of  dimensions  of  the  original 
pattern  space,  to  which  the  number  of  components  of  vector  z is  immediately 
related,  has  great  importance.  Part  II  in  work  devoted  to  methods 

of  reducing  the  dimensions  which  are  part  of  the  problem  of  reducing  data. 

The  defects  mentioned  are  also  partially  associated  with  the  fact  that 
implementation  of  the  decision  rule  is  based  on  histograms.  As  can  be  seen 
from  the  example  presented  in  sub  section  3>  it  does  not  pay  to  reserve  space 
in  the  computer  memory  to  record  histogram  coordinates  for  all  combinations 
of  index  z,  since  many  of  them  may  not  be  useful.  In  this  situation  it  is 
better  to  use  a method  based  on  alternatives  to  form  histograms  for  individuaJ 
classes  in  the  study  cycle: 

a)  increase  the  actual  set  of  numbers  composed  of  z pairs;  the  index 
and  the  frequency  corresponding  to  it,  if  there  occurs  an  observation  be- 
longing to  a cell  not  yet  delimited; 
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b)  correct  the  frequency,  if  the  observation  belongs  to  one  of  the  cells 
earlier  delimited. 

If  the  number  of  blocks  in  the  first  logic  layer  is  greater  than  the 
number  L of  classes  (redun^incy ),  a possibility  exists  in  this  situation 
of  assigning  more  than  one  block  per  class,  observations  of  which  are  par- 

g * 

ticularly  unfavorably  extended  with  several  modes).  In  this  case  the 

redundant  blocks  have  the  nature  of  universal  reserve  blocks  and  can  be 

9 

arbitrarily  assigned  to  chosen  classes.  Among  other  things,  this  is  an  important 
virtue  in  relation  to  segmental  linear  classifier^  in  which  a prtori  deter- 
mination of  the  number  of  modes  alloted  to  individual  classes  is  required. 

A full  study  cycle  of  the  classifier  must  include  an  estimate  of 

3 

two  parameters: 

a)  , on  which  the  number  of  reduced  space  dimension* depends,  and 

b ) S on  which  the  generalizing  properties  of  the  classifier  depend. 

We  make  the  initial  choice  of  p^rameterr  as  a function  of  the  property 

of  observation  taking  part  in  the  study  cycle.  If  we  do  not  assume  the 
e 

existence  of  probability  distribution,  we  have  at  our  disposition  a series 
of  observations,  particularly  the  study  cycle,  and  it  is  possible  to 
choose  f close  to  or  equal  to  unity.  In  the  case  where  information  of  a 
s’atistical  nature  is  associate^ observations,  an  estimate  of  the  classifier 
Parameter  must  be  carried  out  from  the  angle  of  extrapolation  of  the  properties 
of  the  classes  on  the  basis  of  known  observations.  This  means  that  it  is 
necessary  to  choose  ( < 1 and  to  aim  at  an  €S finiofe.  of  £ 

opt 

Since  the  decision  quality  depends  on  the  p^remater  £ and  has  an 
extreme  /4,5  J,  it  is  possible ^empirical  methods,  such  asA final  increment 
alg  jri tnm  £,3  to  ts+i  . In  the  case  of  application  of 
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methods  based  on  stochastic  approximation  algorithm^  it  seems  that  the 
limitation  of  a sign  correction,  normally  based  on  computation  of  a derivative, 
can  be  replaced  by  an  investigation  of  the  error  structure,  the  relationship 
between  netural  and  errenous  decisions.  In  addition  it  is  also  possible  to  use 
one  of  the  methods  of  random  search  to  find  the  minimum  P( c ) . In  this 
way  the  problem  of  optimizing  the  decision  rule  will  be  reduced  to  a problem 
of  minimizing  the  criterion  function  dependent  on  estimated  perameterS', 

The  generalizing  and  selective  properties  of  the  classifier  are 

3 

associated  with  the  choice  of  parameter  £ and  the  number  of  connections 

between  logic  layers.  In  the  research  we  carried  out  we  used  all  possible 

connections  between  layers,  even  though  this  is  not  possible  under  real 

conditions.  Elaboration  of  a method  of  choosing  suitable  connection^ for  the 

3 

purpose  of  reducing  their  number  (or  of  increasing  cell  volume)  &a>\  improve 

/*-  ^ ~r 

classifier  properties  as  well  simplify  structure.-  thus  the  proper  method 
of  reducing  the  dimensions  of  original  pattens  space,  and  then  the  choice 
of  connections  between  logic  layers,  can  thoroughly  simplify  the  structure 
of  a classifier  and  preserve  good  decision  quality. 

From  the  above  considerations  and  from  the  research  carried  out 
it  is  obvious  that  great  possibilities  exist  for  improved  pattern  classification, 
both  in  a wide  range  of  alteration  in  the  properties  of  observation  sets  and 
in  the  set  number  of  inputs  in  the  logic  unit  of  the  classifier.  In  addition 
experiments  carried  out  show  that  it  is  generally  possible,  without  great 
difficulty,  to  obtain  correct  classification  of  all  observations  known  from 
this  study  cycle.  This  property  is  of  great  importance  for  deterministic 
situation* where  final  observation  sets  occur. 

IT.  POSSIBLITIES  OF  PRACTICAL  IMPLEMENTATION  OF  THE  ALGORITHM  /f 


1.  Introduction 
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The  construction  of  a logic  classifier,  as  a completely  independent 
device. is  complex  and  requires  further  solution  of  a system  guided  by  the 
study  of  individual  layers.  It  seems  most  reasonable  and  simple  to  construct 

C<_ 

a classifier  in  the  form  of  hybrid  apparatus  (fig.  8)  composed  ot^  pre- 
processing layer  and  adaptive  networks  (the  classifier  proper),  with  a 
computer  (as  the  guide  system  in  the  study  cycle).  Such  factors  as  the  <‘o 
following  support  this  solution: 

1)  ease  in  constructing  a classifier  as  equipment  cooperating  with  a 
computer, 

2)  possibility  of  using  iterative  methods  to  adjust  the  pre-processing 
layer,  and 

3)  the  use  of  binary  signals  to  choose  to  structure  of  the  logic  unit 
of  a classifier. 

After  the  study  cycle,  the  classifier  can  independently  implement  the 
work  eye le . 

We  shall  not  concerns  ourselves  with  the  technical  implementation 
of  a pre-processing  layer.  It  is  constructed  of  linear  elements,  and  the 
difficulties  which  can  occur  during  implementation  are  generally  known 
/ 7,8.7.  An  interesting  example  of  the  solution  of  this  element  in 
computer  technolog^a#!,  making  cooperation  with  a computer  possible,  has 
been  worked  out  and  produced  in  the  Bionic  Laboratory  of  the  Institute  of 
Applied  Cybernetics  of  PAN  £ Polish  Acamedy  of  Sciences  7/e;. 

2.  Implementation  of  the  tasic  Ufa  it  of  a Jpgic^  Classifier 
2.1  Description  of  ASL-3  Module 

The  concept  for  a technological  solution  of  the  basic  unit  of  a 
classifier  given  below  has  been  worked  out  on  the  basis  of  the  universal  ASL- 6 

Sk 


module.  The  ASL-3  module  (Fig.  9)  is  a modification  of  module  VJ-3  T9]  and  repre- 
sents the  basic  structural  unit  of  a classifier.  In  contains  an  adaptive  logic 
network  with  3 inputs  and  1 output.  The  universal  logic  network  is  formed  of  />^9 6 
8 gates  through  which  all  combinations  of  signats  from  input  bn_2 » ^n-1*  b^ 

(where  n is  an  index  of  the  last  position  in  the  binary  expansion  of  the  number) 
e.g.,  y1  = bib2 ...bn_2  ^ b^)  an< 3 control  signals  from  flip-flops  LPO,  ) LP1, 

...  LP7  are  fed.  The  outputs  from  the  gates  are  connected  throggh  a logic  sum 
with  the  output  of  the  module.  Just  as  for  module  W-3,  a special  input  s serves  /497 
to  expand  the  area  of  length  of  binary  numbers  introduced  into  the  module.  The 
method  of  using  input  s will  be  illustrated  as  an  example. 

LP  flip-flops,  guided  by  matrix  M and  an  address  network  (used  exclusively 
in  the  study  cycle)  take  part  in  the  control  memory  of  the  module.  The  diagram 
of  matrix  M is  shown  in  Fig.  9b  in  the  shape  of  a black  box,  and  its  description 
is  included  in  Table  2.  The  class  properties  are  extrapolated  with  the  aid  of 
matrix  M on  the  basis  of  each  observation  which  occurred  in  the  study  cycle. 

These  extrapolation  nodes  (points  on  the  coordinate  axis)  are  delimited  bv 
observations  through  the  medium  of  signal  Adr  from  the  address  network  in  the 
module.  The  signal  e,  a counterpart  of  parameter  £ , defines  the  environment 
in  which  property  extrapolation  takes  place.  In  particular  signal  e = 1 means 
simple  memorization  of  observations  which  occurred  in  the  study  cycle.  In  case 
e^  1,  the  observations  are  memorized  along  with  the  distinctive  environment 
(Fig.  %)  • Assignment  of  observations  to  a class  is  determined  by  signal  dj. 

To  illustrate  the  properties  discussed,  let  us  look  at  an  example  where 
e = 3.  Occurrence  of  observations  at  the  model  input  causes  matrix  M,  to 
which  the  signals  Adr  = 1 (from  observations)  and  dj  = 1 are  addressed,  gives 
a signal 
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LP=1  and  signal  ig=2  going  in  two  directions.  Under  the  effect  of  signal 

4 A4r 

r p in  a neighboring  matrix  (occurring  as  r^=2)  with  ABR=0(d ^=l,e=3 ), 

we  get  LP=1  and  r0=l.  The  signal  r =1  in  the  next  matrix  causes  LP=1, 

^ 2 

r^=0  (see  last  line  in  table  2).  The  lack  of  a signal  r^  means  that  the 
process  of  extrapolation  from  the  observation? was  completed.  In  this  way 
the  condition  of  the  flip-flops  created  at  the  end  of  the  study  cycle 
guarantees  implementation  of  a definite  auxiliary  function  by  the  logic 
network.  In  the  work  cycle,  the  lack  of  signal  e(e=0)  protects  flip- 
flops  LPi,i=0, 1, . . .,7  from  changes  in  condition  (see  first  line  in  table 
2). 

2.2.  Construction  of  t,ogic  Network 

With  suitable  connection  of  modules  we  get  a complex  network  in  which 
the  number  of  binary  input1’ for  each  component  of  the  vector  of  observation 
can  be  greater  than  3.  Fig.  10  shows  an  example  of  a bilay€Y~ 
logic  network  constructed  of  ASL-3  modules.  This  network  can  serve  as  a 
binary  classifier  for  patterns  y=(y^,y^,y^,y^ ) . The  values  of  signals 
corresponding  to  individual  pattern  components  are  binary  numbers  and 
have  the  following  shape 


Mi 


/W 


li  = bi  , 

y2  = b i b2b2b4, 

y 3 ~ b i b2b3b4.bs , 

y*  = b\ b2b3, 


(1) 


where:  b =0  or  1, j=l,2, . . . , 5 • 
J 
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The  S matrices  in  the  network  presented  work  as  commutators.  The 


module,  in  which  a logic  unit  is  found  oft  an  additional  input,  will  be 
chosen  as  a function  of  the  condition  of  signals  b^and  bg  at  their  input 
(we  shall  explain  this  for  component  y^).  This  unit  opens  a route  for 
further  signals  b^,b,  ,b  of  component  y . In  this  way  the  breakdown  of  the 

i 4 5 1 

t 

permissible  set  of  values  of  component  y^  into  four  subsets,  with  the  aid  of 
matrix  S,  corresponds  to  the  choice  of  module 


00000  01  000  10000  11  000 
00001  01  001  10  001  11 001 

00  111  01  111  10  111  11(11 


(2) 


^hch  sub  set  (2)  is  a set  of  values  of  arguments  for  a suitable  module. 

A second  layer  of  the  logic  unit  of  a (decision)  classifier  can  be 


composed  of  exactly  the  same  ASL-3  as  the  first  layer.  The  input  signal 


of  the  second  layer  is  a binary  vector  z formed  of  output  signals  from 
tne, (see  figure  10,z=(z. ,z  -z  ,z  ).  The  basic  difference  between  the 

/V  1 2 3 4 

first  and  second  layers  is  the  following.  For  the  first  layer  every 


CL 
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observation  y can  be  treated  as  a point  dimensional  space  of  features, 

the  coordinates  of  which  assume  discrete  valued  from  a certain  section.  In 
the  case  of  the  second  layer  the  vector  of  signal  z can  be  treated  as  a point 
in  KqL- dimensional  binary  space  (apex  of  a unit  Sopt >rblvcl^).  In  addition 
the  output  signal  of  the  second  layer  is  simultaneously  the  output  signal  for 
the  entire  network. 

The  structure  of  the  second  layer  will  be  chosen  at  e=l,  in 
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conformity  with  rule  (l)(part  l),  and  the  quality  of  the  decisions  organized 
by  the  f block,  apices  is  determined  with  the  aid  of  criterion  (2)  (part  I). 

3*  Final  Considerations 

An  important  feature  in  the  technological  solution  of  the  basic  unit 
of  a classifier  is  the  possibility  of  achieving  it  in  the  form  of  a logic 
network  with  a modular  structure.  This  means  that  both  network  layers  can 
be  composed  and  expanded  with  the  same  ASL-3  modules. 

The  construction  of  the  ASL-3  module  as  a computer  system  does  not 
present  any  serious  difficulties.  The  two-phase  work  of  computer  systemSis 
associated  with  rather  slight  requirements  in  element  tolerance  and  causes 
them  to  be  suitable  for  technological  realization,  particularly  in  the 
form  of  self-contained  circuits.  In  addition  the  possibility  of  using 
self-contained  circuits  for  this  purpose  brings  about  other  advantages, 
such  as  great  reliability,  small  production  costs,  miniaturization,  great 
rapidity  of  computer  system  activity,  etc. 

One  short  combing  in  the  solution  presented  for  the  basic  unit  of 
a classifier  is  the  expanded  structure  of  the  ASL-3  module  (effect  of 
universality).  Redundant  construction  of  the  decision  layer  occurs  in 
connection  with  this,  since  in  this  case  the  modules  are  used  only  for 
e=l.  In  addition  the  lower  flexibility  (than  for  a programmed  version) 
in  choosing  the  extrapolation  peraraeter  e can  cause  greater  classification 
errors  in  certain  situations. 

In  the  general  case  the  basic  unit  can  work  as  an  adaptive  logic 

/? 

network,  and  particularly  as  a pattern  classifier.  Working  out  the  linear 
(threshold ) element,  in  contridistinction  to  logic  systems,  can  present 
some  difficulties.  These  difficulties  occur  mainly  from  the  need  to  use 
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-the-  A 

non-typical  solutions  ii^  construction^altered  weights  of  linear  elements 
Z7,8j.  The  use  of  computer  technology  for  this  pux-pose  turns  out  to 
be  very  advantageous  from  many  points  of  view  TO-  In  general  classification 
systems  in  which  the  linear  element  is  one  of  the  main  components ( 6 . ^ . , 

segmental  linear  classifiers,  threshold  layer  classifiers)  are  more  expedient 
in  program  implementation  tn*n  in  systems  implementation*,. 
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Detailed  description  of  the  present  algorithm  and  an  example  of 
measuring  set  coveriag  can  be  found  in  J'j  J . 

In  general  £ can  be  a vector  of  component  ( £ j , c ). 

An  example  of  the  method  of  choosing  connections  based  on  the 
algorithm  of  pre+processing  layer  adaptation  is  discussed  in  point 
2.1.  given  in  hi  • 

Attention  should  be  called  to  the  fact  that  the  output  signal  r^ 
of  one  matrix  is  the  input  signal  r^  of  a neighboring  matrix. 
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Summary 

The  classification  algorithm  described  in  part  I can  be  used  in  the  case  of  incomplete  initial  informa- 
tion. From  the  functional  point  of  view  in  the  classifier  there  are  distinguished:  the  preprocessing  layer, 
the  transformation  layer  and  the  decision  layer.  During  learning  a rule  was  applied  according  to  which 
particular  classifier  layers  are  subsequently  adjusted. 

The  main  feature  of  the  algorithm  concerns  the  partitioning  of  pattern  space  into  oells.  Then  for  each 
class  a histogram  is  deviced  being  an  estimate  of  an  unknown  distribution  of  the  probability  of  appearing 
of  the  pattern  from  particular  cells.  The  way  of  building  cells  and  histograms  takes  into  account  the  form 
of  initial  information.  The  details  concerning  the  algorithm  are  explained  on  an  example. 

In  the  part  II  there  was  given  a concept  of  practical  implementation  of  the  classifiers  basic  unit  in  the 
form  of  two  layers  of  adaptive  logical  networks.  Each  layer  of  logical  network  can  be  assembled  of  modelas 
ASL-3.  The  module  ASL-3  (Fig.  9)  is  the  basic  construction  unit  of  the  classifier.  It  is  an  adaptive  logical 
network  with  three  inputs  and  one  output.  In  the  case  of  building  the  classifier  there  exist  a possibility  of 
connecting  it  simply  for  working  with  a computer. 
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